Add Ollama Function Call support #1111

tzolov · 2024-07-24T18:26:01Z

extend the OllamaApi with Tool, ToolCalls and Function and add Tool Role.
make OllamaChatModel extend the AbstractToolCallSupport.
extend the OllamaChatModel to convert the Spring AI messages to OllamaApi messages, including Tools.
OllamaOptions implements FunctionCallingOptions.
add OllamaApiToolFunctionCallIT for testing function calling.
patch the AbstractToolCallSupport#isToolCall to take set of stop resons.
add FunctionCallbackInPromptIT and FunctionCallbackWrapperIT function calling auto-configuration tests.
extend OllamAutoConfiguration to support function calling registration.
add function call tests to OllamChatAutoConfigurationIT.
add OllamaWithOpenAiChatModelIT to OpenAi that uses the OpenAI API to call Ollama.
move buildToolCallConversation and handleToolCalls to parent AbstractToolCallSupport.
update Docs. add Ollama function call docs.
update Ollama diagrams.

Resolves #720

- extend the OllamaApi with Tool, ToolCalls and Function and add Tool Role. - make OllamaChatModel extend the AbstractToolCallSupport. - extend the OllamaChatModel to convert the Spring AI messages to OllamaApi messages, including Tools. - OllamaOptions implements FunctionCallingOptions. - add OllamaApiToolFunctionCallIT for testing function calling. - patch the AbstractToolCallSupport#isToolCall to take set of stop resons. - add FunctionCallbackInPromptIT and FunctionCallbackWrapperIT function calling auto-configuration tests. - extend OllamAutoConfiguration to support function calling registration. - add function call tests to OllamChatAutoConfigurationIT. - add OllamaWithOpenAiChatModelIT to OpenAi that uses the OpenAI API to call Ollama. - move buildToolCallConversation and handleToolCalls to parent AbstractToolCallSupport. - update Docs. add Ollama function call docs. - update Ollama diagrams. Resolves spring-projects#720

ThomasVitale · 2024-07-24T18:29:34Z

models/spring-ai-ollama/src/main/java/org/springframework/ai/ollama/api/OllamaApi.java

+			 * @param parameters The parameters the functions accepts, described as a JSON Schema object. To describe a
+			 * function that accepts no parameters, provide the value {"type": "object", "properties": {}}.
+			 */
+			public record Function(


For parity with the Ollama types, this might be named ToolFunction

It is already nested inside the record Tool, so new ChatRequest.Tool.ToolFunction would read awkardly.

ThomasVitale · 2024-07-24T18:31:46Z

...ollama/src/test/java/org/springframework/ai/ollama/api/tool/OllamaApiToolFunctionCallIT.java

+		assertThat(responseMessage.toolCalls()).isNotNull();
+
+		// Check if the model wanted to call a function
+		if (responseMessage.toolCalls() != null) {


This check shouldn't be needed since the previous assertion makes sure the tool calls are not null.

ThomasVitale · 2024-07-24T18:33:48Z

models/spring-ai-openai/src/main/java/org/springframework/ai/openai/OpenAiChatModel.java

@@ -175,7 +175,7 @@ public ChatResponse call(Prompt prompt) {

 		ChatResponse chatResponse = new ChatResponse(generations, from(completionEntity.getBody(), rateLimit));

-		if (isToolCall(chatResponse, OpenAiApi.ChatCompletionFinishReason.TOOL_CALLS.name())) {
+		if (isToolCall(chatResponse, Set.of(OpenAiApi.ChatCompletionFinishReason.TOOL_CALLS.name(), "stop"))) {


Should this be Set.of(OpenAiApi.ChatCompletionFinishReason.TOOL_CALLS.name(), OpenAiApi.ChatCompletionFinishReason.STOP.name())?

Also, consider adding a comment about why we need the "stop" reason as well.

OpenAiApi.ChatCompletionFinishReason.STOP.name()) seems better

ThomasVitale · 2024-07-24T18:34:14Z

...-ai-openai/src/test/java/org/springframework/ai/openai/chat/OllamaWithOpenAiChatModelIT.java

+@Disabled("For manual smoke testing only.")
+@Testcontainers
+@SpringBootTest(classes = OllamaWithOpenAiChatModelIT.Config.class)
+class OllamaWithOpenAiChatModelIT {


Really nice!

ThomasVitale · 2024-07-24T18:37:04Z

spring-ai-core/src/main/java/org/springframework/ai/chat/model/AbstractToolCallSupport.java

@@ -95,6 +97,20 @@ protected Set<String> handleFunctionCallbackConfigurations(FunctionCallingOption
 		return functionToCall;
 	}

+	protected List<Message> handleToolCalls(Prompt prompt, ChatResponse response) {


It's really nice we could abstract this. It might help in the future if we allow "deconstructing" the function calling workflow and have users calling the functions explicitly.

Indeed I was thinking the same. Maybe the AbstractToolCallSuppor is not the perfect abstraction for direct usage but we will figure it out.

ThomasVitale · 2024-07-24T18:38:40Z

...ollama/src/test/java/org/springframework/ai/ollama/api/tool/OllamaApiToolFunctionCallIT.java

+@Testcontainers
+public class OllamaApiToolFunctionCallIT {
+
+	private static String MODEL = "mistral";


this could be

private static final String MODEL = OllamaModel.MISTRAL.getName();

sounds good

ThomasVitale · 2024-07-24T18:39:46Z

models/spring-ai-ollama/src/test/java/org/springframework/ai/ollama/OllamaChatModelIT.java

@@ -65,7 +64,7 @@ class OllamaChatModelIT {
 	@Container
 	static OllamaContainer ollamaContainer = new OllamaContainer("ollama/ollama:0.1.32");


Consider updating this image to version "0.2.8" as well

ThomasVitale · 2024-07-24T18:39:56Z

...pring-ai-ollama/src/test/java/org/springframework/ai/ollama/OllamaChatModelMultimodalIT.java

@@ -53,7 +53,7 @@ class OllamaChatModelMultimodalIT {
 	@Container
 	static OllamaContainer ollamaContainer = new OllamaContainer("ollama/ollama:0.1.32");


Consider updating this image to version "0.2.8" as well

I updated and it seems there is a regression, the description of the image returned now is totally wrong. I'll merge the PR with the updated versions but we need to circle back and manually test.

ThomasVitale · 2024-07-24T18:40:33Z

...-ai-ollama/src/test/java/org/springframework/ai/ollama/OllamaChatModelFunctionCallingIT.java

+
+	private static final Logger logger = LoggerFactory.getLogger(OllamaChatModelFunctionCallingIT.class);
+
+	private static String MODEL = "mistral";


this could be

private static final String MODEL = OllamaModel.MISTRAL.getName();

ThomasVitale · 2024-07-24T18:50:58Z

Super!

markpollack · 2024-07-24T20:31:27Z

addressed comments and made other minor cosmetic changes, merged in d53876a

ThomasVitale reviewed Jul 24, 2024

View reviewed changes

markpollack self-assigned this Jul 24, 2024

markpollack closed this Jul 24, 2024

markpollack added this to the 1.0.0-M2 milestone Jul 24, 2024

		@@ -65,7 +64,7 @@ class OllamaChatModelIT {
		@Container
		static OllamaContainer ollamaContainer = new OllamaContainer("ollama/ollama:0.1.32");

		@@ -53,7 +53,7 @@ class OllamaChatModelMultimodalIT {
		@Container
		static OllamaContainer ollamaContainer = new OllamaContainer("ollama/ollama:0.1.32");


		private static final Logger logger = LoggerFactory.getLogger(OllamaChatModelFunctionCallingIT.class);

		private static String MODEL = "mistral";

Add Ollama Function Call support #1111

Add Ollama Function Call support #1111

Uh oh!

Conversation

tzolov commented Jul 24, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ThomasVitale commented Jul 24, 2024

Uh oh!

markpollack commented Jul 24, 2024

Uh oh!

Uh oh!